Phase detection autofocus noise reduction

ABSTRACT

Certain aspects relate to systems and techniques for using imaging pixels (that is, non-phase detection pixels) for performing noise reduction on phase detection autofocus. Advantageously, this can provide for more accurate phase detection autofocus and also to optimized processor usage for performing phase detection. The phase difference detection pixels are provided to obtain a phase difference detection signal indicating a shift direction (defocus direction) and a shift amount (defocus amount) of image focus, and analysis of imaging pixel values can be used to estimate a level of focus of an in-focus region of interest and to limit the identified phase difference accordingly.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 14/865,503, filed on Sep. 25, 2015, entitled “PHASE DETECTIONAUTOFOCUS NOISE REDUCTION,” which is related to U.S. patent applicationSer. No. 14/864,153, filed on Sep. 24, 2015, entitled “MASK-LESS PHASEDETECTION AUTOFOCUS” and U.S. patent application Ser. No. 14/865,629,filed on Sep. 25, 2015, entitled “PHASE DETECTION AUTOFOCUS ARITHMETIC,”the contents of which are hereby incorporated by reference herein.

TECHNICAL FIELD

The systems and methods disclosed herein are directed to phase detectionautofocus, and, more particularly, to analyzing imaging pixels to reducenoise in the phase detection autofocus process.

BACKGROUND

Some image capture devices use phase difference detection sensors (whichmay also be referred to as “pixels”) to perform autofocus. On-sensorphase difference detection works by interspersing phase differencedetection pixels between imaging pixels, typically arranged in repeatingsparse patterns of left and right pixels. The system detects phasedifferences between signals generated by different phase differencedetection pixels, for example between an image generated from left pixeldata and an image generated from right pixel data. The detected phasedifferences can be used to perform autofocus.

Phase detection autofocus operates faster than contrast-based autofocus.Many current implementations place a metal mask over the image sensor tocreate left and right phase detection pixels, resulting in less lightreaching the masked pixels. Because the output of phase detection pixelshas lower brightness than the output of normal image capturing pixels,the phase difference detection pixels create noticeable artifacts incaptured images that require correction. By placing the phase detectionpixels individually amidst imaging pixels, the system can interpolatevalues for the phase detection pixels.

Masked pixels are used in pairs. When the scene is out of focus, thephase detection pixel mask phase shifts the incoming light slightly. Thedistance between phase detection pixels, combined with their relativeshifts, can be convolved to give a determination of roughly how far theoptical assembly needs to move the lens to bring the scene into focus.

SUMMARY

In signal processing, noise is generally refers to unwanted and possiblyunknown modifications or errors that a signal may suffer during capture,storage, transmission, processing, or conversion. More specifically, inautofocus contexts, noise can refer to inaccurate determinations offocus conditions. In some phase detection autofocus systems, for examplewith sparsely positioned phase detection pixels (for example, havingfive or more imaging pixels located between phase detection pixels in apair), target scenes with high frequency patterns may cause theautofocus system to falsely identify a defocus condition with the targetscene is actually in focus. In signal processing, this phenomena, whichoccurs when a high frequency signal is sampled at a low frequency, isalso referred to as aliasing or aliasing noise.

The aforementioned problems, among others, are addressed in someembodiments by the phase detection autofocus systems and techniquesdescribed herein that use values received from imaging pixels to reducenoise of the autofocus process. For example, values from imaging pixelscan be analyzed to provide information about high frequency details inthe target scene and to identify a sharpest edge in the image data ofthe target scene. The phase detection process can be limited to aspecific range of possible defocus conditions based on a sharpnessmetric of the sharpest edge. As a result, processing time and resourcesfor performing autofocus can be optimized. Further, any identifieddefocus condition may not exceed a largest defocus possible forproducing the sharpest edge.

One innovation includes an imaging apparatus including a plurality ofdiodes each configured to capture one of image information representinga target scene, the image information received from a first subset ofthe plurality of diodes, or phase detection information, the phasedetection information received from a second subset of the plurality ofdiodes; and a processor configured with instructions to analyze theimage information identify a region of the image information that ismore in-focus than another region of the image information, anddetermine a focus value of the region, identify a bounding phasedifference value corresponding to the focus value, generate at leastfirst and second images based on the phase detection information, anddetermine a phase difference between the first and second images, andcalculate an autofocus adjustment based at least partly on thedetermined phase difference and at least partly on a range defined bythe bounding phase difference value.

The following are non-limiting examples of some features and embodimentsof such imaging apparatuses. For example, the processor can beconfigured to determine the focus value by identifying a sharpest edgerepresented by the image information. The processor can be configured todetermine the focus value based at least partly on a sharpness value ofthe sharpest edge. The imaging apparatus can further include a maskingelement positioned above each of the second subset of the plurality ofdiodes to block a portion of light propagating from the target scene soeach of the second subset of the plurality of diodes only collects thelight propagating from the target scene in a specific direction. Theimaging apparatus can further include a microlens positioned above atleast two adjacent diodes of the second subset of the plurality ofdiodes, the microlens formed to pass a first portion of lightpropagating from the target scene to a first of the at least twoadjacent diodes and a second portion of light propagating from thetarget scene to a second of the at least two adjacent diodes, the firstportion propagating in a first direction and the second portionpropagating in a second direction. The plurality of diodes can be asemiconductor substrate having a two-dimensional matrix of the pluralityof diodes. Optical elements associated with a first half of the secondsubset of the plurality of diodes can be configured such that each diodeof the first half of the second subset of the plurality of diodes onlycollects light propagating from the target scene in a first direction togenerate a first half of the phase detection information. Opticalelements associated with a second half of the second subset of theplurality of diodes can be configured such that each diode of the secondhalf of the second subset of the plurality of diodes only collects lightpropagating from the target scene in a second direction to generate asecond half of the phase detection information.

Another innovation includes a processor configured with instructions forperforming a process for phase detection autofocus, the processincluding accessing image information representing a target scene from afirst plurality of imaging diodes; analyzing the image information toevaluate a focus value of at least one region; identifying a boundingphase difference value corresponding to the focus value; generating atleast first and second images based on phase detection informationaccessed from a second plurality of imaging diodes; determining a phasedifference between the first and second images; and calculating anautofocus adjustment based at least partly on the determined phasedifference and at least partly on a range defined by the bounding phasedifference value.

The following are non-limiting examples of some features and embodimentsof such processors. For example, identifying the bounding phasedifference value can include determining whether the determined phasedifference value falls within a range defined by the bounding phasedifference value; and in response to determining that the determinedphase difference value exceeds the range, outputting the bounding phasedifference value as a phase difference between the first and secondimages. Outputting the bounding phase difference value as a phasedifference can be further performed in response to determining that theprocess for phase detection autofocus is in a monitoring state. Themonitoring state can occur after the process for phase detectionautofocus has identified an in-focus condition and moved a lens assemblyassociated with the plurality of imaging sensing elements and the atleast one phase detection sensing element to a focus positioncorresponding to the in-focus condition. Identifying the phasedifference can include analyzing the phase difference information withina range defined by the bounding phase difference value. The process canfurther include partitioning the image information into a plurality ofregions and analyzing the plurality of regions to evaluate a pluralityof focus values each corresponding to one of the plurality of regions.The process can further include identifying a plurality of boundingphase differences value each based on one of the plurality of focusvalues. The process can further include combining the plurality ofbounding phase difference values to produce the bounding phasedifference value used for calculating the autofocus adjustment. Theprocess can further include selecting one of the plurality of boundingphase difference values of at least some of the plurality of regions tooutput as the bounding phase difference value used for calculating theautofocus adjustment.

Another innovation includes a process for phase detection autofocusincluding accessing image information from a plurality of imagingdiodes, the image information representing a target scene; accessingphase detection information received from a plurality of phase detectiondiodes; determining, based on the image information, informationrepresenting at least one depth discontinuity in the target scene;identifying a pair of the plurality of phase detection diodes located onopposing sides of the at least one depth discontinuity; and calculatingan autofocus adjustment based on phase detection information receivedfrom a subset of the plurality of phase detection diodes excluding thepair.

The following are non-limiting examples of some features and embodimentsof such processes. For example, determining the information representingthe at least one depth discontinuity in the target scene can includeinterpolating, from the image information, a first center pixel value ata location of a first phase detection diode in the pair and a secondcenter pixel value at a location of a second phase detection diode inthe pair; calculating a first disparity value between the first centerpixel value and a value received from the first phase detection diode;calculating a second disparity value between the second center pixelvalue and a value received from the second phase detection diode; andcomparing a difference between the first disparity and the seconddisparity to a threshold. The process can further include excluding thepair from the subset of the plurality of phase detection diodes inresponse to determining that the difference is greater than thethreshold. Interpolating the first center pixel value can furtherinclude accessing image information received from a 5×5 neighborhood ofthe plurality of imaging diodes around a location of the first phasedetection diode. The process can further include computing thedifference between the first disparity and the second disparity.Determining the information representing the at least one depthdiscontinuity in the target scene can include performing edge detectionon the image information. Calculating an autofocus adjustment caninclude using phase detection information received from the subset ofthe plurality of phase detection diodes to generate a pair ofhalf-images; determining a phase difference between the half-images; andcalculating the autofocus adjustment based on the phase difference.

Another innovation includes an imaging apparatus including means forcapturing image information representing a target scene; a plurality ofmeans for capturing phase detection information; means for identifying aregion of the image information that is more in-focus than anotherregion of the image information and determining a focus value of theregion; means for identifying a bounding phase difference valuecorresponding to the focus value; means for generating at least firstand second images based on the phase detection information; means fordetermining a phase difference between the first and second images, andmeans for calculating an autofocus adjustment based at least partly onthe determined phase difference and at least partly on a range definedby the bounding phase difference value.

The following are non-limiting examples of some features and embodimentsof such imaging apparatuses. For example, the imaging apparatus canfurther include means for partitioning the image information into aplurality of regions, wherein the means for identifying the boundingphase difference value can be configured to identify a plurality ofbounding phase difference values each corresponding to one of theplurality of regions. The imaging apparatus can further include meansfor combining the plurality of bounding phase difference values of atleast some of the plurality of regions to produce the bounding phasedifference value used for calculating the autofocus adjustment. Theimaging apparatus can further include means for selecting one of theplurality of bounding phase difference values of at least some of theplurality of regions to output as the bounding phase difference valueused for calculating the autofocus adjustment. The means for generatingat least first and second images based on the phase detectioninformation can include an image sensor including logic. The means forgenerating at least first and second images based on the phase detectioninformation can include an image signal processor.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed aspects will hereinafter be described in conjunction withthe appended drawings and appendices, provided to illustrate and not tolimit the disclosed aspects, wherein like designations denote likeelements.

FIG. 1A depicts a schematic view of an example multi-diode microlens forobtaining phase detection autofocus information.

FIGS. 1B and 1C depict schematic views of example masked diodes forobtaining phase detection autofocus information.

FIG. 2A depicts an example ray trace of light entering a pair of phasedetection diodes.

FIG. 2B depicts an example ray trace of light entering a pair of phasedetection diodes.

FIG. 2C depicts an example ray trace of light entering a pair of phasedetection diodes.

FIG. 2D depicts an example ray trace of light entering a pair of maskedphase detection diodes.

FIGS. 3A and 3B depict graphical representations of example methods forcalculating a center pixel value.

FIG. 3C depicts a graphical representation of an example method forcalculating a virtual pixel value.

FIG. 3D illustrates a graphical representation of offset of variousphase detection images.

FIG. 4A illustrates a flowchart of one embodiment of a process forperforming a noise-reduced phase detection autofocus process asdescribed herein.

FIG. 4B illustrates a flowchart of another embodiment of a process forperforming a noise-reduced phase detection autofocus process asdescribed herein.

FIG. 5 depicts a high-level overview of an example phase detectionautofocus process using a sensor having the multi-diode microlenses.

FIG. 6 depicts a schematic block diagram illustrating an example of animaging system equipped with the phase detection autofocus devices andtechniques.

DETAILED DESCRIPTION Introduction

Embodiments of the disclosure relate to systems and techniques for usingimaging pixels (that is, non-phase detection pixels) for performingnoise reduction on phase detection autofocus. Advantageously, this canprovide for more accurate phase detection autofocus and also tooptimized processor usage for performing phase detection. The phasedifference detection pixels are provided to obtain a phase differencedetection signal indicating a shift direction (defocus direction) and ashift amount (defocus amount) of image focus, and analysis of imagingpixel values can be used to estimate a level of focus of an in-focusregion of interest and to limit the identified phase differenceaccordingly.

Image data received from imaging pixels can be used, for example, toexclude incorrect phase difference calculations based on calculating amaximum allowable phase difference value from analysis of the imagedata. If the region of neighboring pixels around a particular phasedetection pixel shows strong or sharp edges, for example based on acontrast analysis, this can indicate that the image is close to thefocus point. Accordingly, the noise reduction method may bound themaximum possible phase difference value that can be identified by thephase detection pixels. This maximum may be used to omit wrong phasedetection calculations, for example, phase differences that exceed thecalculated maximum. The maximum can also be used to optimize the phasedetection algorithm so that it will not need to perform a full phasedetection search. In one example the range of phase detection searchingcan be limited based on the maximum possible phase difference. Inanother example, the autofocus system can directly report the image tobe in focus (for example, by reporting a PD phase equal to 0).

Sparse phase detection diode patterns can suffer from problems due toaliasing, as the diodes in a phase detection pair can be located onopposite sides of a sharp edge. Phase detection autofocus can beparticularly susceptible to errors due to aliasing in circumstances inwhich the target scene includes high frequency patterns. Further,aliasing can be more problematic in the monitoring state of an autofocusprocess; in the monitoring state the imaging system has alreadyconverged to an in-focus position and the target scene is monitored forchanges that may require re-positioning the imaging system to a newin-focus (converged) position. This is due to the fact that highfrequencies appear only when the target scene is in focus or close tothe in-focus point. Accordingly, analysis of data received from imagingpixels to identify high frequency patterns and/or edges having athreshold level of sharpness can be used to eliminate or compensate forerrors that might otherwise be introduced in the phase detection processdue to aliasing.

The present phase detection autofocus technique can use imaging pixelsin a neighborhood of a single phase detection pixel, illustrated belowin FIG. 1 as a right phase detection pixel, to perform noise reductionon the phase detection process. Other examples can use a left pixel.Pixel values from neighboring pixels in various sizes of regions, forexample 3×3, 5×5, 7×7, and so on can provide valuable informationregarding the usefulness of a particular phase detection pixel forautofocus purposes.

Some examples can interpolate a center pixel value at the location ofeach phase detection pixel in a pair for performing noise reductioncalculations. The center pixel value can be interpolated from fouradjacent green pixels or from neighboring pixels in various sizes ofregions, for example 5×5, 7×7, and the like. In some embodiments, thecenter interpolation process can be part of a process to compensate fora value at the phase detection pixel location in the final image, forexample similar to defective pixel correction. Various implementationscan include on-sensor logic for center interpolation or rely on anexternal ISP to perform the center interpolation process. The autofocussystem can calculate a disparity value between the interpolated centerpixel value and the phase detection pixel value for each of the phasedetection pixels in the pair. If a difference between these disparityvalues is greater than a threshold, this can indicate that the phasedetection pixels in the pair are likely located on opposing sides of anedge, and the pair of phase detection pixels may be excluded fromautofocus calculations.

In another embodiment, the autofocus system can use data received fromimaging pixels to perform edge detection, for example all imaging pixelsor a subset of the imaging pixels in a predetermined neighborhood arounda pair of phase detection pixels. The autofocus system can then comparethe locations of edges to the known locations of the phase detectionpixels to determine whether the phase detection pixels in the pair arelikely located on opposing sides of an edge, and the pair of phasedetection pixels may be excluded from autofocus calculations. Othersuitable techniques for determining whether the phase detection pixelsin a pair lie on opposite sides of a depth discontinuity, for exampleanalysis of intensity variation in image pixel data, can also be used toexclude data received from such pairs of phase detection pixels fromautofocus adjustment calculations. Excluding data from a pair of phasedetection pixels located on opposite sides of an edge can include usingdata received from a subset of phase detection pixels not including theidentified pair when generating a pair of half-images usable for phasedifference detection.

Though discussed herein primarily in the context of masked phasedetection pixels, the noise reduction techniques described herein arealso applicable to mask-less phase detection pixels, for examplereceiving phase information due to microlens formation.

Various embodiments will be described below in conjunction with thedrawings for purposes of illustration. It should be appreciated thatmany other implementations of the disclosed concepts are possible, andvarious advantages can be achieved with the disclosed implementations.Headings are included herein for reference and to aid in locatingvarious sections. These headings are not intended to limit the scope ofthe concepts described with respect thereto. Such concepts may haveapplicability throughout the entire specification.

Overview of Example Phase Detection Microlens and Color FilterArrangements

FIG. 1A depicts a schematic view of an example sensor portion 100including a multi-diode microlens 110 to generate phase data. The sensorportion 100 includes single-diode microlenses 105, multi-diode microlens110, a filter array 108 including a plurality of color filters 115, andphotodiodes (“diodes”) 120A-120D arranged in a sensor array 107.Multi-diode microlens 110 is sized and positioned such that incominglight from a target scene propagates through the multi-diode microlens110 before falling incident on the includes diodes 120B, 120C of thesensor array 107, which are covered by the multi-diode microlens 110such that light pa.

The diodes can be, for example, photodiodes formed in a semiconductorsubstrate, for example in a complementary metal-oxide semiconductor(CMOS) image sensor. As used herein, diode refers to a single unit ofany material, semiconductor, sensor element or other device thatconverts incident light into current. The term “pixel” as used hereincan refer to a single diode in the context of its sensing functionalitydue to adjacent optical elements such as color filters or microlenses.Accordingly, although “pixel” generally may refer to a display pictureelement, a “pixel” as used herein may refer to a sensor (for example, aphotodiode) that receives light and generates a signal which if renderedon a display, may be displayed as a point in an image captured by thesensor (and a plurality of other sensors). The individual units orsensing elements of an array of sensors, for example in a CMOS orcharge-coupled (CCD) device, can also be referred to as sensels.

Color filters 115 act as wavelength-selective pass filters and split (orfilter) incoming light in the visible range into red, green, and blueranges. The incoming light is, for example, from a target scene. Thelight is “split” by allowing only certain selected wavelengths to passthrough the color filters 115, the filtering of light based on theconfiguration of the color filters 115. Various types of color filters115 may be used. As illustrated in FIG. 1A, the split light is receivedby the dedicated red, green, or blue diodes 120A-120D. Although red,blue, and green color filters are commonly used, in other embodimentsthe color filters can vary according to the color channel requirementsof the captured image data, for example including ultraviolet, infrared,or near-infrared pass filters.

Each single-diode microlens 105 is positioned over a single color filter115 and a single diode 120A, 120D. Diodes 120A, 120D accordingly provideimaging pixel information. The color filters under the single-diodemicrolenses in a microlens array 106 can be positioned according to theBayer pattern in some embodiments. Multi-diode microlens 110 ispositioned over two adjacent color filters 115B, 115C and twocorresponding adjacent diodes 120B, 120C. Diodes 120B, 120C accordinglyprovide phase detection pixel information by diode 120B receiving lightentering multi-diode microlens 110 in a first direction and diode 120Creceiving light entering multi-diode microlens 110 in a seconddirection. The color filters under the multi-diode microlenses in themicrolens array 106 can be selected to pass the same wavelength of lightin some embodiments.

As used herein, “over” and “above” refer to the position of a structure(for example, a color filter or lens) such that light incident from atarget scene propagates through the structure before it reaches (or isincident on) another structure. To illustrate, the microlens array 106is positioned above the color filter array, which is positioned abovethe diodes 120A-120D. Accordingly, light from the target scene firstpasses through the microlens array, then the color filter array, andfinally is incident on the diodes 115A-115D.

FIG. 1A depicts (dashed) line 130 which should be understood is not aphysical structure but rather is depicted to illustrate the phasedetection capabilities provided by multi-diode microlens 110. The line130 passes through the optical center of the multi-diode microlens 110and is orthogonal to a plane formed by the color filter array of colorfilters 115A-115D. Where multi-diode microlens 110 is a 2×1 microlens,the multi-diode microlens 110 is formed such that light L(x) incident ina first direction, that is, entering the multi-diode microlens 110 fromone side of the line 130, is collected in a first diode 120B. Lightincident in a second direction, that is, entering the multi-diodemicrolens 110 from the other side of the line 130, is collected in asecond diode 120C. Accordingly, data received from diodes 120B, 120C canbe used for phase detection. Where multi-diode microlens is a 2×2microlens, the multi-diode microlens 110 is formed such that light L(x)incident in four directions, with a direction considered as lightpassing through a quarter of the multi-diode microlens 110, is incidenton four diodes.

FIGS. 1B and 1C depict schematic views of example masked diodes forobtaining phase detection autofocus information. FIG. 1B illustrates anexample arrangement 100B having masks 125A, 125B positioned below colorfilters 115, while FIG. 1C illustrates an example arrangement 100Chaving masks 125A, 125B positioned above color filters 115. Light fromthe target scene passes through a primary focusing lens assembly (notillustrated) before passing through masks 125A, 125B, microlenses 115,and color filters and then falls incident on a pair of phase detectiondiodes 120E, 120F. Phase detection diodes 120E, 120F may be adjacent toone another or may be spaced apart by one or more diodes, and may be inthe same row or column or in different rows or columns, depending uponthe design considerations of the image sensor.

The masks 125A, 125B allow only light incident in a specific direction(as depicted, from one side of dashed line 130 through the opticalcenter of the single-diode microlens 105) to be selectively collected inthe diodes 120E, 120F. For example, in order to generate a phasedifference, light-blocking masks 125A, 125B are disposed over two diodes120E, 120F in an opposite direction to produce a pair, and tens ofthousands of pairs can be arranged in a two-dimensional matrix of animage sensor to obtain data for phase difference autofocus. The amountof light incident on the diodes 120E, 120F is reduced by 50% relative toimaging diodes by the half-apertures created by masks 125A, 125B withrespect to the optical axes of the single-diode microlenses 105.

FIGS. 2A-2C depict example ray traces of light traveling through a mainlens 250 then through a multi-diode microlens 110 before fallingincident on a pair of phase detection diodes 120B, 120C. It will beappreciated that the dimensions of the main lens 250 and the multi-diodemicrolens 110 are not shown to scale. The diameter of the multi-diodemicrolens 110 can be approximately equal to the distance spanning twoadjacent diodes of an image sensor, while the diameter of the main lens250 can be equal to or greater than the width (the distance along a rowor column of diodes) of the image sensor.

Specifically, FIG. 2A depicts an example ray trace of an in-focuscondition, FIG. 2B depicts an example ray trace of a front-focuscondition, and FIG. 2C depicts an example ray trace of a back-focuscondition. Light travels from a point 260 in a target scene, travelsthrough lens 250 for focusing the target scene onto an image sensorincluding the phase detection diodes 120B, 120C, and passes through themulti-diode microlens 110 before falling incident the phase detectiondiodes 120B, 120C. As illustrated, diode 120B receives light from a leftdirection L(x) of the main lens 250 and diode 120C receives light from aright direction R(x) of the main lens 250. In some embodiments lightfrom the left direction L(x) can be light from a left half (depicted asthe lower half in the illustration of FIGS. 2A-1C) of the main lens 250and light from the right direction R(x) can be light from a right half(depicted as the upper half in the illustration of FIGS. 2A-1C) of themain lens 250. Accordingly, a number of phase detection diodesinterleaved with imaging diodes across the image sensor can be used toextract left and right images that are offset from a center imagecaptured by the imaging diodes. Rather than right and left, otherembodiments can use up and down images, diagonal images, or acombination of left/right, up/down, and diagonal images for calculatingautofocus adjustments.

When the image is in focus, the left rays L(x) and right rays R(x)converge at the plane of the phase detection diodes 115B, 115C. Asillustrated in FIGS. 1C and 2C, in front and back defocus positions therays converge before and after the plane of the diodes, respectively. Asdescribed above, signals from the phase detection diodes can be used togenerate left and right images that are offset from the center image infront or back defocus positions, and the offset amount can be used todetermine an autofocus adjustment for the main lens 250. The main lens250 can be moved forward (toward the image sensor) or backward (awayfrom the image sensor) depending on whether the focal point is in frontof the subject (closer to the image sensor), or behind the subject(farther away from the image sensor). Because the autofocus process canfigure out both the direction and amount of movement for main lens 250,phase-difference autofocus can focus very quickly.

FIG. 2D depicts an example ray trace of light traveling through a mainlens (not illustrated) then through apertures formed by phase detectionmasks 125A, 12B before falling incident on the pair of phase detectiondiodes 120F, 120E. FIG. 2D depicts a back-focus condition, as the raysfrom the left L(x) and right R(x) sides of the main lens converge behindthe plane of the image sensor. As illustrated, light from the right R(x)side of the main lens falls incident on diode 120F, while light from theleft L(x) side of the main lens falls incident on diode 120E due to thehalf-apertures created by phase detection masks 125A, 12B.

Overview of Example Multi-Image Phase Detection Autofocus

FIG. 3A depicts a graphical representation of an example method forcalculating a center pixel value for a masked phase detection diode. Asused herein, “center pixel value” refers to a color and/or brightnessvalue interpolated at the location of a phase detection diode usingsignals received from one or more nearby imaging diodes. Although theexample of FIG. 3A depicts a masked phase detection diode, otherimplementations may use the multi-diode microlens described above toproduce phase data.

As shown in the illustration of FIG. 3A, in one embodiment a centerpixel value can be interpolated at the location 305 of a phase detectiondiode based on four diagonally-adjacent green pixel values. Thisapproach can be used in some implementations having a green color filterat the location 305 over the phase detection diode. In other examplesthe center value can be calculated from neighboring pixels of any colorin various sizes of regions, for example 5×5, 7×7, and the like.

In some embodiments, the phase detection diode may have a clear colorfilter or no color filter disposed over the diode. Although anotherfilter, for example an infrared cut-off filter, may still be disposedover the diode, the phase detection diode under the clear color filteror no color filter receives visible light that has not been colorfiltered. Accordingly, the brightness of the light incident on the diodecan be increased relative to a diode having a wavelength-selective passfilter, which may be beneficial for provide phase data in low lightsettings due to the reduction of light incident on the diode by thehalf-aperture. In such embodiments, the center pixel value can beinterpolated based on neighboring pixel values of all colors, forexample values received from all eight diodes in a 3×3 neighborhoodsurrounding the location of the phase detection diode. In one example, alinear weighting function can be used to determine the center pixelvalue from the eight diodes in the 3×3 neighborhood.

A center image can be constructed from center pixel values from a numberof locations across the image sensor. Half-images can be constructedfrom values received from phase detection diodes at a number oflocations across the image sensor.

FIG. 3B depicts a graphical representation of an example method forcalculating a center pixel value for a pair of phase detection diodesunder a multi-diode microlens 110. In the illustrated example, bothdiodes under the multi-diode microlens 110 are below green colorfilters. Accordingly, five green pixel values in a 3×4 neighborhoodsurrounding the multi-diode microlens 110 are used to calculate thecenter pixel value. In other embodiments, green pixel values fromdiffering neighborhoods can be used, for example two green values from a5×1 neighborhood along the row of the multi-diode microlens, greenvalues from a 5×6 neighborhood, etc.

In another embodiment, both diodes under the multi-diode microlens 110can be below clear color filters. In such embodiments, the center pixelvalue can be determined based on neighboring pixel values of all colors,for example values received from all ten diodes in the 3×4 neighborhoodsurrounding the location of the multi-diode microlens 110. In oneexample, a linear weighting function can be used to determine the centerpixel value from the ten diodes in the 3×4 neighborhood.

The same center pixel value can be used for both diodes under themulti-diode microlens 110, that is, half-images generated based partlyon values received from both diodes under the multi-diode microlens 110can be compared to a center image generated based partly on the centerpixel value.

FIG. 3C depicts a graphical representation of an example method forcalculating a virtual pixel value. In the illustrated example, a valuereceived from a right phase detection sensel (a diode such as diodes120C, 120E described above receiving light from a right R(x) side of afocusing lens assembly) is subtracted from the interpolated center pixelvalue to produce a virtual phase detection pixel value, depicted asvirtual left phase detection pixel value L_(v). Right phase detectionpixel value R and virtual left phase detection pixel value L_(v) form apair of opposing phase detection pixels that can be used in someembodiments to generate half-images together with a number of otherright and virtual left phase detection pixel values. The half-images canbe compared to one another for determining phase difference usable fordetermining an autofocus adjustment.

It will be appreciated that similar techniques can be used to generatevirtual right, up, down, and diagonal phase detection pixel values atthe location of an opposing phase detection pixel in a pair.

FIG. 3D illustrates a graphical representation of an offset continuum300 depicting the offset various phase detection pixel values L, L_(v),R, and R_(v) relative to a center pixel value C when the imaging systemused to capture the images is in a defocus condition. The offsetindicates an amount of defocus and a direction of defocus compared to afocus point. As will be understood, in an in-focus condition there islittle or no offset, and the pixel values C, L, L_(v), R, and R_(v)would be located at the same point on the offset continuum 300.

The defocus condition and its effect on the C, L, L_(v), R, and R_(v),pixel values is illustrated by the circle of confusion diagrams 301-305above the offset continuum 300. A circle of confusion, also referred toas a blur spot, is an optical spot caused by a cone of light rays notcoming to a focus point at the plane of the image sensor when imaging apoint source, for example occurring in the focus conditions illustratedin FIGS. 2B and 2C. A whole blur spot 301 is illustrated correspondingto the center pixel value, which represents light incident on a senselthrough an unmasked aperture. A crescent-shaped blur spot 302,illustrated in comparison to the shape of the blur spot 301, is incidenton a sensel under a mask blocking a right half (or approximate half) ofthe aperture above the sensel, thereby producing the left phasedetection pixel value L. Similarly, a crescent-shaped blur spot 303,illustrated in comparison to the shape of the blur spot 301, is incidenton a sensel under a mask blocking a left half (or approximate half) ofthe aperture above the sensel, thereby producing the right phasedetection pixel value R. The blur spot 304 of the virtual left 4 pixel,having been calculated as the subtraction of the crescent-shaped blurspot 303 of the right phase detection pixel from the whole blur spot 301of the pixel value C, is approximately marquise-shaped or lens-shaped.Similarly, the blur spot 305 of the virtual right R_(v) pixel, havingbeen calculated as the subtraction of the crescent-shaped blur spot 302of the left phase detection pixel from the whole blur spot 301 of thepixel value C, is approximately marquise-shaped or lens-shaped.

As described above, the center pixel value C can be used, together witha number of other calculated center pixel values, to generate a centerimage. Phase detection pixel values L, L_(v), R, and R_(v) can be used,together with a number of a number of other left, virtual left, right,and virtual right phase detection pixel values, respectively, togenerate half-images usable for detecting phase difference.

The center image C is illustrated as having no offset (located at thezero point along the offset continuum 300). The left L and right R pixelvalues are illustrated as having an negative and positive offsets,respectively, from the center pixel value C, based on the center of massof the corresponding one of crescent-shaped blur spots 302, 303. Typicalphase detection autofocus processes use the disparity between the left Land right R pixel values to calculate an autofocus adjustment. However,in the phase detection autofocus processes described herein, theautofocus adjustment can be calculated based on disparity between imagesgenerated from a number of left pixel values L and center pixel valuesC, right pixel values R and center pixel values C, or based on bothdisparities.

Performing phase detection between a center image and a left or rightimage generated based on values corresponding to the same sensellocations can result in a less noisy phase detection autofocus processwhen compare to performing autofocus using a left image and a rightimage generated base on values received from different sensel locations(for example, from a pair of left and right phase detection senselspositioned with a number of imaging sensels in between) due to reducingthe effects of aliasing. Further, in some embodiments the resolution ofthe phase detection process can be doubled by using disparities betweenboth the center image and the left image and the center image and theright image. In other embodiments the number of phase detection senselson the sensor can be reduced, thereby producing less artifacts incaptured image data.

Further, the virtual left L_(v) and virtual right R_(v) pixel values areillustrated as having negative and positive offsets, respectively, fromthe center image C. The distance between the center of mass of centerblur spot 301 and the center of mass of the crescent-shaped blur spots302, 303 corresponding to the actual left and right phase detectionpixels is greater than the distance between the center of mass of centerblur spot 301 and the center of mass of the lens-shaped blur spots 204,305 corresponding to the virtual left L_(v) and virtual right R_(v)pixels. Thus, the offset of the virtual left L_(v) and virtual rightR_(v) pixel values from the center pixel value C can be smaller than theoffset of corresponding left L and right R pixel values from the centerpixel value C. For example, the masked sensels used to produce the leftL and right R pixel values may have masks that do not create an apertureperfectly centered on the optical axis of the microlens above the maskedsensel, for example by being sized or positioned such that the effectivereduced aperture is not exactly aligned with the microlens optical axis.As another example, the center of mass of the blur generated by a left Lpixel value may be offset from the center of mass of the center pixelvalue C by more or less than the center of mass of the blur generated bythe corresponding virtual right R_(v) pixel value.

In some implementations, phase detection correlation can be performedusing any or all using the following combinations of images:

(1) the virtual left image with the right image or virtual right image,(2) the virtual right image with the left image or virtual left image,and(3) the center image and any of the other four images.

Some implementations can create a combined right image based on theright virtual phase detection pixel values and the right actual phasedetection pixel values and also create a combined left image based onthe left virtual phase detection pixel values and the left actual phasedetection pixel values; such implementations can then perform left-rightimage correlation on the combined images, thereby doubling the amount ofphase detection information compared to a typical left-right correlationperformed without virtual phase detection information. In one example ofa robust phase detection algorithm, correlation can be searched foracross all possible combinations of the five images illustrated in FIG.3D.

In one embodiment, generation of the center image and half-images forphase detection can be performed on-sensor. In other implementations thecenter image and half-images can be generated by an image signalprocessor based on values received from phase detection diodes andnearby imaging diodes.

Overview of Example Phase Detection Noise Reduction Techniques

Sparse phase detection diode patterns can suffer from problems due toaliasing, as the diodes in a phase detection pair can be located onopposite sides of an edge. Phase detection autofocus can be particularlysusceptible to errors due to aliasing in circumstances in which thetarget scene includes high frequency patterns. Further, aliasing can bemore problematic in the monitoring state of an autofocus process; in themonitoring state the imaging system has already converged to an in-focusposition and the target scene is monitored for changes that may requirere-positioning the imaging system to a new in-focus (converged)position. This is due to the fact that high frequencies appear only whenthe target scene is in focus or close to the in-focus point.Accordingly, analysis of data received from imaging pixels to identifyhigh frequency patterns and/or edges having a threshold level ofsharpness can be used to eliminate or compensate for errors that mightotherwise be introduced in the phase detection process due to aliasing.

FIG. 4A illustrates a flowchart of one embodiment of a process forperforming a noise-reduced phase detection autofocus process 400A asdescribed herein. The process phase detection autofocus process 400A canbe used to optimize the phase detection search range by limiting thepossible disparity values based on analysis of imaging pixel data. Someembodiments of the phase detection autofocus process 400A can beperformed on-sensor, and other embodiments can be performed with the useof an image signal processor.

At block 405, the process 400A can analyze regular image pixels. Thiscan include, for example, receiving data representing values of lightcollected in diodes of an image sensor that are not dedicated to phasedetection. At block 410, the process 400A can evaluate the maximum blur.As used herein, “maximum blur” refers to the level of blur in image datathat can be used to limit phase detection for autofocus. The maximumblur can be the level of blurriness of a most-in focus portion of theimage, for example a most in-focus region within a focus region ofinterest.

In some embodiments, the maximum blur can be estimated using contrastfocus statistics blocks. For example, in contrast-based autofocustechniques, part of the analysis can be implemented in a HW block tooutput a focus value in a certain region of interest (ROI). Suchtechniques may get multiple focus values for different locations, forexample as a grid mapped to part or the whole image, or may get aseparate focus value configured for multiple windows.

In one implementation, the maximum blur can be a value representing anamount of blur of a sharpest edge identified in the data from theimaging pixels. Accordingly, the blocks 405, 410 of analyzing the imagedata from imaging pixel and evaluating the maximum blur can involveperforming edge detection on the image data, identifying a sharpestedge, and determining a level of blur of the sharpest edge.

In one implementation, blocks 405 and 410 can involve the followingsteps. A horizontal Fourier Transform or fast Fourier Transform can beperformed on the image data from the imaging pixels. Next, the process400A can search for a highest frequency in the resulting data having anenergy above a predetermined threshold. The process 400A can apply twolinear filters, for example a finite impulse response filter or aninfinite impulse response filter. Each of the two filters can beconfigured to have a different band pass. A first filter of the twofilters can be configured with a high frequency band pass (for example,0.5-0.2 cycles/pixel) and a second filter of the two filters can beconfigured with a low frequency band pass (for example, 0.2-0.1cycles/pixel).

In one implementation, if the energy in the data resulting from the highfrequency band pass is above the predetermined threshold, the process400A can estimate that the maximum blur is 5 pixels, else if the energyin the data resulting from the low frequency band pass is above thepredetermined threshold then the process 400A can estimate that themaximum blur is 10 pixels, and otherwise the process 400A cannot limitphase detection by a maximum blur.

At block 415, the process 400A can evaluate a phase difference valuePD_MAX corresponding to the evaluated maximum blur. This valuerepresents a bounding phase difference value, and valid phase differencevalues can be expected to fall within a range between positive andnegative values of the bounding phase difference value. Accordingly, thephase difference can be limited to not exceed the estimated level ofdefocus of an in-focus region of interest of the image as determined bythe analysis of the data received from imaging diodes.

At block 420, the process 400A can perform phase difference detectionacross the range defined by the phase difference value corresponding tothe blur of the in-focus region of interest, depicted as [−PD_MAX,+PD_MAX]. As discussed below with respect to FIG. 5, phase differencedetection can include the following steps. Left and right channels (alsoreferred to herein as left and right half-images or left and rightimages) can be extracted from data received from left and right phasedetection diodes, and the left and right channels can be corrected foruniformity, for example using a gain map similar to correction of lensshading. The process 400A can then find the horizontal correlationbetween the left and right channels. This can involve searching within arange of positive and negative pixel shifts, for example −3 pixels to +3pixels, and this search range can be scaled according to the maximumblur as discussed above. For each pixel shift, the process 400A canmeasure the sum of absolute difference (SAD) to obtain a SAD vector, andcan interpolate the SAD vector to find an optimal correlation pointbetween the channels. The phase detection offset determined by thecorrelation is then converted into a defocus lens position change n, forexample using linear mapping, polynomial function, multi-segment linear,a look-up table, or any other suitable function. The determined phasedifference can be used for calculating an autofocus adjustment, forexample by being used to determine a direction and amount of movement ofa focusing lens to position the focusing lens at an in-focus position.

FIG. 4B illustrates a flowchart of another embodiment of a process 400Bfor performing a noise-reduced phase detection autofocus process asdescribed herein. The process phase detection autofocus process 400B canbe used to compensate aliasing errors that can occur in the focusmonitoring phase of autofocus by ignoring disparity values that exceed amaximum determined based on analysis of imaging pixel data. Someembodiments of the phase detection autofocus process 400B can beperformed on-sensor, and other embodiments can be performed with the useof an image signal processor.

At block 420, the process 400B performs phase difference detectionacross the full range of available differences received from phasedetection diodes. Simultaneously or sequentially, blocks 405, 410, and415 described above with respect to FIG. 4A are performed to identify aphase difference range corresponding to the level of blur in an in-focusregion of interest of data received from imaging pixels.

The phase difference range derived from blocks 405, 410, and 415 isused, at block 425, to perform a range check on the phase differencedetection performed at block 420. The range check identifies whether theidentified phase difference (depicted as PD in block 420) falls withinthe range of [−PD_MAX, +PD_MAX].

If the identified phase difference falls within the determined phasedifference range, then the process 400B moves to block 430 to use theidentified phase difference to calculate an autofocus adjustment. Afterthe lens and/or sensor of the imaging system are moved to the in-focusposition corresponding to the determined autofocus adjustment, theprocess 400B indicates a transition to the monitoring state ofautofocus. In some embodiments, this can involve periodic repetition ofthe process 400A of FIG. 4A. In other embodiments, this can involvere-initializing blocks 420 and 405 of process 400B.

If the identified phase difference does not within the determined phasedifference range, then the process 400B moves to block 435 indicatingthan an aliasing error is suspected. As described above, aliasing cancause the identified phase difference value to exceed an actual level ofblur of an in-focus portion of the image.

At block 440, the process 400B identifies whether the autofocus processis in the monitoring state. For example, the process 400B can determinewhether the imaging system has already been moved to (or was determinedto already be in) an in-focus position. If the autofocus process is notin the monitoring state, then the 400B moves to block 445 to ignore theidentified phase difference. At this point, blocks 420 and 405 can bere-initialized.

If the autofocus process is in the monitoring state, then the 400B movesto block 450 to clip the identified phase difference to PD_MAX, thephase difference determined to correspond to the level of blur in thein-focus region of interest. The process 400B can then re-initializeblocks 420 and 405 or can switch, in some embodiments, to the focusmonitoring process 400A of FIG. 4A.

Some imaging apparatuses may implement process 400A and/or 400B todetermine a bounding phase difference value separately for multipleregions of image information. For example, image information can bepartitioned into a grid across the field of view (for example a 5×5grid). In another example, the imaging apparatus can determine abounding phase difference value on each line of image data separatelyand then combine the bounding phase difference values or select one ofthe bounding phase difference values to use for determining the validphase difference range for autofocus calculations. The imaging apparatuscan also remove the noise (phase detection sensel pairs havingout-of-range disparity values) for each region separately and/ordisqualify shift phase detection sensel values that that are out ofbound for specific lines/regions.

In another embodiment of a noise reduction phase detection technique,the imaging apparatus can interpolate a center pixel value at thelocation of each phase detection pixel in a pair for performing noisereduction calculations. The center pixel value can be interpolated fromfour adjacent green pixels or from neighboring pixels in various sizesof regions, for example 5×5, 7×7, and the like. In some embodiments, thecenter interpolation process can be part of a process to compensate fora value at the phase detection pixel location in the final image, forexample similar to defective pixel correction. Various implementationscan include on-sensor logic for center interpolation or rely on anexternal ISP to perform the center interpolation process. The imagingapparatus can calculate a disparity value between the interpolatedcenter pixel value and the phase detection pixel value for each of thephase detection pixels in the pair. If a difference between thesedisparity values is greater than a threshold, this can indicate that thephase detection pixels in the pair are likely located on opposing sidesof an edge, and the pair of phase detection pixels may be excluded fromautofocus calculations.

In another embodiment, the imaging apparatus can use data received fromimaging pixels to perform edge detection, for example all imaging pixelsor a subset of the imaging pixels in a predetermined neighborhood arounda pair of phase detection pixels. The imaging apparatus can then comparethe locations of edges to the known locations of the phase detectionpixels to determine whether the phase detection pixels in the pair arelikely located on opposing sides of an edge, and the pair of phasedetection pixels may be excluded from autofocus calculations. Othersuitable techniques for determining whether the phase detection pixelsin a pair lie on opposite sides of a depth discontinuity, for exampleanalysis of intensity variation in image pixel data, can also be used toexclude data received from such pairs of phase detection pixels fromautofocus adjustment calculations. Excluding data from a pair of phasedetection pixels located on opposite sides of an edge can include usingdata received from a subset of phase detection pixels not including theidentified pair when generating a pair of half-images usable for phasedifference detection.

Overview of Example Phase Detection Autofocus Process

FIG. 5 depicts a high-level overview of an example phase detectionautofocus process 500 that can implement the multi-image phase detectionand noise reduction processes described herein. In one embodiment, theprocess 500 can be performed on-sensor. In other implementations, theprocess 500 can involve one or more processors, for example image signalprocessor 620 of FIG. 6. Light representing the target scene 505 ispassed through the lens assembly 510 and received by the image sensor,where half-image samples 515 are produced as described above. Thehalf-image samples can include a right image, virtual right image, leftimage, virtual left image, and a center image.

The lens assembly 510 can be modeled as a linear low-pass filter with asymmetric impulse response, wherein the impulse response (also referredto as the point spread function) of the lens assembly 510 is of arectangular shape with a width parameter proportional to the distancebetween the sensor and the image plane. The scene is “in focus” when thesensor is in the image plane, that is, in the plane where all rays froma single point at the scene converge into a single point. As shown inFIG. 2, the half-image samples can save two images containing onlyinformation from the phase detection pixels. The half-images can beconsidered as convolutions of the target scene with left and right (or,in other examples, up and down) impulse responses of the lens assembly510. In sensor embodiments using the techniques of FIGS. 3A-3D, morepartial images can be saved.

A focus function calculator 520 applies a cross-correlation function tothe partial images to determine disparity. As described above,cross-correlation can be searched for across one or more pairs of imagesselected from the right image, virtual right image, left image, virtualleft image, and a center image. Further, cross-correlation may belimited to a range determined by analyzing data received from imagingdiodes to estimate a level of focus of an in-focus region of interest ofthe data.

The cross-correlation function of the left and right impulse responsesof the lens assembly 510 can be approximately symmetric and unimodal,however due to the nature of the target scene 505, the cross-correlationfunction as applied to the left and right captured images may have oneor more false local maxima. Various approaches can be used to identifythe true maximum of the cross-correlation function. The result of thecross-correlation function is provided as feedback to the autofocuscontrol 525, which can be used to drive a lens actuator to move theprimary focusing lens assembly 510 to a desired focus position. Otherembodiments may use a stationary primary focusing lens assembly and movethe image sensor to the desired focus position. Accordingly, in thephase detection autofocus process 500, focusing is equivalent tosearching for the cross-correlation function maximum. This is a fastprocess that can be done quickly enough to provide focus adjustment foreach frame at typical frame rates, for example at 30 frames per second,and thus can be used to provide smooth autofocusing for video capture.Some implementations combine phase detection autofocus withcontrast-based autofocus techniques, for example to increase accuracy.

When the primary focusing lens assembly and/or image sensor are in thedesired focus position, the image sensor can capture in-focus imagingpixel information and phase detection pixel information and, asdescribed above, calculate and interpolate color values for the phasedetection pixels. The imaging pixel values and determined phasedetection pixel values can be output for demosaicking and, optionally,other image processing techniques to generate a final image of thetarget scene.

Overview of Example Phase Detection Autofocus Process

FIG. 6 illustrates a high-level schematic block diagram of an embodimentof an image capture device 600 having multispectral iris authenticationcapabilities, the image capture device 600 having a set of componentsincluding an image signal processor 620 linked to a phase detectionautofocus camera 615. The image signal processor 620 is also incommunication with a working memory 605, memory 630, and deviceprocessor 650, which in turn is in communication with storage module 610and an optional electronic display 625.

Image capture device 600 may be a portable personal computing devicesuch as a mobile phone, digital camera, tablet computer, personaldigital assistant, or the like. There are many portable computingdevices in which using the phase detection autofocus techniques asdescribed herein would provide advantages. Image capture device 600 mayalso be a stationary computing device or any device in which themultispectral iris verification techniques would be advantageous. Aplurality of applications may be available to the user on image capturedevice 600. These applications may include traditional photographic andvideo applications as well as data storage applications and networkapplications.

The image capture device 600 includes phase detection autofocus camera615 for capturing external images. The phase detection autofocus camera615 can include an image sensor having multi-diode microlenses and colorfilters, or masked phase detection pixels, arranged according to theembodiments described above. The phase detection autofocus camera 615can also have a primary focusing mechanism positionable based at leastpartly on data received from the image signal processor 620 to producean in-focus image of the target scene. In some embodiments, the primaryfocusing mechanism can be a movable lens assembly positioned to passlight from the target scene to the sensor. In some embodiments, theprimary focusing mechanism can be a mechanism for moving the sensor.

The sensor of the phase detection autofocus camera 615 can havedifferent processing functionalities in different implementations. Inone implementation, the sensor may not process any data, and the imagesignal processor 620 may perform all needed data processing. In anotherimplementation, the sensor may be capable of extracting phase detectionpixels, for example into a separate Mobile Industry Processor Interface(MIPI) channel. Further, the sensor may additionally be capable ofinterpolating center pixel values, for example in a RAW channel. In someimplementations the sensor may additionally be capable of interpolatingcenter pixel values, for example in a normal channel, and may be able toprocess the whole phase detection calculation internally (on-sensor).For example, the sensor may include analog circuitry for performingsums, subtractions, and/or comparisons of values received from diodes.An imaging apparatus as described herein may include an image sensorcapable of performing all phase detection calculations or an imagesensor capable of performing some or no processing together with animage signal processor 620 and/or device processor 650.

The image signal processor 620 may be configured to perform variousprocessing operations on received image data in order to execute phasedetection autofocus and image processing techniques. Image signalprocessor 620 may be a general purpose processing unit or a processorspecially designed for imaging applications. Examples of imageprocessing operations include demosaicking, white balance, cross talkreduction, cropping, scaling (e.g., to a different resolution), imagestitching, image format conversion, color interpolation, colorprocessing, image filtering (e.g., spatial image filtering), lensartifact or defect correction, etc. The image signal processor 620 canalso control image capture parameters such as autofocus andauto-exposure. Image signal processor 620 may, in some embodiments,comprise a plurality of processors. Image signal processor 620 may beone or more dedicated image signal processors (ISPs) or a softwareimplementation of a processor. In some embodiments, the image signalprocessor 620 may be optional for phase detection operations, as some orall of the phase detection operations can be performed on the imagesensor.

As shown, the image signal processor 620 is connected to a memory 630and a working memory 605. In the illustrated embodiment, the memory 630stores capture control module 635, phase detection autofocus module 640,and operating system module 645. The modules of the memory 630 includeinstructions that configure the image signal processor 620 of deviceprocessor 650 to perform various image processing and device managementtasks. Working memory 605 may be used by image signal processor 620 tostore a working set of processor instructions contained in the modulesof memory. Alternatively, working memory 605 may also be used by imagesignal processor 620 to store dynamic data created during the operationof image capture device 600.

As mentioned above, the image signal processor 620 is configured byseveral modules stored in the memories. The capture control module 635may include instructions that configure the image signal processor 620to adjust the focus position of phase detection autofocus camera 615,for example in response to instructions generated during a phasedetection autofocus technique. Capture control module 635 may furtherinclude instructions that control the overall image capture functions ofthe image capture device 600. For example, capture control module 635may include instructions that call subroutines to configure the imagesignal processor 620 to capture multispectral image data including oneor more frames of a target scene using the phase detection autofocuscamera 615. In one embodiment, capture control module 635 may call thephase detection autofocus module 240 to calculate lens or sensormovement needed to achieve a desired autofocus position and output theneeded movement to the imaging processor 220. Capture control module 635may call the phase detection autofocus module 240 to interpolate colorvalues for pixels positioned beneath multi-pixel microlenses.

Accordingly, phase detection autofocus module 640 can store instructionsfor executing phase detection autofocus, for example according toprocesses 400A, 400B, and 500 described above with respect to FIGS. 4A,4B, and 5. Phase detection autofocus module 640 can also storeinstructions for calculating center pixel values and virtual phasedetection pixel values as described above with respect to FIGS. 3A-3C.

Operating system module 645 configures the image signal processor 620 tomanage the working memory 605 and the processing resources of imagecapture device 600. For example, operating system module 645 may includedevice drivers to manage hardware resources such as the phase detectionautofocus camera 615. Therefore, in some embodiments, instructionscontained in the image processing modules discussed above may notinteract with these hardware resources directly, but instead interactthrough standard subroutines or APIs located in operating systemcomponent 650. Instructions within operating system 645 may theninteract directly with these hardware components. Operating systemmodule 645 may further configure the image signal processor 620 to shareinformation with device processor 650.

Device processor 650 may be configured to control the display 625 todisplay the captured image, or a preview of the captured image, to auser. The display 625 may be external to the imaging device 200 or maybe part of the imaging device 200. The display 625 may also beconfigured to provide a view finder displaying a preview image for a useprior to capturing an image, for example to assist the user in aligningthe image sensor field of view with the user's eye, or may be configuredto display a captured image stored in memory or recently captured by theuser. The display 625 may comprise an LCD, LED, or OLED screen, and mayimplement touch sensitive technologies.

Device processor 650 may write data to storage module 610, for exampledata representing captured images and data generated during phasedetection and/or pixel value calculation. While storage module 610 isrepresented schematically as a traditional disk device, storage module610 may be configured as any storage media device. For example, thestorage module 610 may include a disk drive, such as an optical diskdrive or magneto-optical disk drive, or a solid state memory such as aFLASH memory, RAM, ROM, and/or EEPROM. The storage module 610 can alsoinclude multiple memory units, and any one of the memory units may beconfigured to be within the image capture device 600, or may be externalto the image capture device 600. For example, the storage module 610 mayinclude a ROM memory containing system program instructions storedwithin the image capture device 600. The storage module 610 may alsoinclude memory cards or high speed memories configured to store capturedimages which may be removable from the camera. The storage module 610can also be external to image capture device 600, and in one exampleimage capture device 600 may wirelessly transmit data to the storagemodule 610, for example over a network connection. In such embodiments,storage module 610 may be a server or other remote computing device.

Although FIG. 6 depicts an image capture device 600 having separatecomponents to include a processor, imaging sensor, and memory, oneskilled in the art would recognize that these separate components may becombined in a variety of ways to achieve particular design objectives.For example, in an alternative embodiment, the memory components may becombined with processor components, for example to save cost and/or toimprove performance.

Additionally, although FIG. 6 illustrates two memory components,including memory 630 comprising several modules and a separate memorycomponent comprising a working memory 605, one with skill in the artwould recognize several embodiments utilizing different memoryarchitectures. For example, a design may utilize ROM or static RAMmemory for the storage of processor instructions implementing themodules contained in memory 630. The processor instructions may beloaded into RAM to facilitate execution by the image signal processor620. For example, working memory 605 may comprise RAM memory, withinstructions loaded into working memory 605 before execution by theimage signal processor 620.

Implementing Systems and Terminology

Implementations disclosed herein provide systems, methods and apparatusfor using values received from imaging sensing elements to reduce noiseof a phase detection autofocus process. One skilled in the art willrecognize that these embodiments may be implemented in hardware,software, firmware, or any combination thereof

In some embodiments, the circuits, processes, and systems discussedabove may be utilized in a wireless communication device. The wirelesscommunication device may be a kind of electronic device used towirelessly communicate with other electronic devices. Examples ofwireless communication devices include cellular telephones, smartphones, Personal Digital Assistants (PDAs), e-readers, gaming systems,music players, netbooks, wireless modems, laptop computers, tabletdevices, etc.

The wireless communication device may include one or more image sensors,two or more image signal processors, a memory including instructions ormodules for carrying out the process discussed above. The device mayalso have data, a processor loading instructions and/or data frommemory, one or more communication interfaces, one or more input devices,one or more output devices such as a display device and a powersource/interface. The wireless communication device may additionallyinclude a transmitter and a receiver. The transmitter and receiver maybe jointly referred to as a transceiver. The transceiver may be coupledto one or more antennas for transmitting and/or receiving wirelesssignals.

The wireless communication device may wirelessly connect to anotherelectronic device (e.g., base station). A wireless communication devicemay alternatively be referred to as a mobile device, a mobile station, asubscriber station, a user equipment (UE), a remote station, an accessterminal, a mobile terminal, a terminal, a user terminal, a subscriberunit, etc. Examples of wireless communication devices include laptop ordesktop computers, cellular phones, smart phones, wireless modems,e-readers, tablet devices, gaming systems, etc. Wireless communicationdevices may operate in accordance with one or more industry standardssuch as the 3rd Generation Partnership Project (3GPP). Thus, the generalterm “wireless communication device” may include wireless communicationdevices described with varying nomenclatures according to industrystandards (e.g., access terminal, user equipment (UE), remote terminal,etc.).

The functions described herein may be stored as one or more instructionson a processor-readable or computer-readable medium. The term“computer-readable medium” refers to any available medium that can beaccessed by a computer or processor. By way of example, and notlimitation, such a medium may comprise RAM, ROM, EEPROM, flash memory,CD-ROM or other optical disk storage, magnetic disk storage or othermagnetic storage devices, or any other medium that can be used to storedesired program code in the form of instructions or data structures andthat can be accessed by a computer. Disk and disc, as used herein,includes compact disc (CD), laser disc, optical disc, digital versatiledisc (DVD), floppy disk and Blu-ray® disc where disks usually reproducedata magnetically, while discs reproduce data optically with lasers. Itshould be noted that a computer-readable medium may be tangible andnon-transitory. The term “computer-program product” refers to acomputing device or processor in combination with code or instructions(e.g., a “program”) that may be executed, processed or computed by thecomputing device or processor. As used herein, the term “code” may referto software, instructions, code or data that is/are executable by acomputing device or processor.

The methods disclosed herein comprise one or more steps or actions forachieving the described method. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isrequired for proper operation of the method that is being described, theorder and/or use of specific steps and/or actions may be modifiedwithout departing from the scope of the claims.

It should be noted that the terms “couple,” “coupling,” “coupled” orother variations of the word couple as used herein may indicate eitheran indirect connection or a direct connection. For example, if a firstcomponent is “coupled” to a second component, the first component may beeither indirectly connected to the second component or directlyconnected to the second component. As used herein, the term “plurality”denotes two or more. For example, a plurality of components indicatestwo or more components.

The term “determining” encompasses a wide variety of actions and,therefore, “determining” can include calculating, computing, processing,deriving, investigating, looking up (e.g., looking up in a table, adatabase or another data structure), ascertaining and the like. Also,“determining” can include receiving (e.g., receiving information),accessing (e.g., accessing data in a memory) and the like. Also,“determining” can include resolving, selecting, choosing, establishingand the like.

The phrase “based on” does not mean “based only on,” unless expresslyspecified otherwise. In other words, the phrase “based on” describesboth “based only on” and “based at least on.”

In the foregoing description, specific details are given to provide athorough understanding of the examples. However, it will be understoodby one of ordinary skill in the art that the examples may be practicedwithout these specific details. For example, electricalcomponents/devices may be shown in block diagrams in order not toobscure the examples in unnecessary detail. In other instances, suchcomponents, other structures and techniques may be shown in detail tofurther explain the examples.

The previous description of the disclosed implementations is provided toenable any person skilled in the art to make or use the presentinvention. Various modifications to these implementations will bereadily apparent to those skilled in the art, and the generic principlesdefined herein may be applied to other implementations without departingfrom the spirit or scope of the invention. Thus, the present inventionis not intended to be limited to the implementations shown herein but isto be accorded the widest scope consistent with the principles and novelfeatures disclosed herein.

What is claimed is:
 1. An imaging system comprising: a plurality ofdiodes, a first subset of the plurality of diodes configured to captureimage information representing a scene; a second subset of the pluralityof diodes configured to capture phase detection information; and aprocessor configured with instructions to analyze the image informationto identify a level of blurriness of a region of interest, determine aphase difference between at least first and second images generatedbased on the phase detection information, determine whether the phasedifference exceeds the level of blurriness, and in response todetermining that the phase difference does not exceed the level ofblurriness, calculate an autofocus adjustment based on the phasedifference.
 2. The imaging system of claim 1, further comprising: animage sensor comprising the plurality of diodes; a lens positioned tofocus light representing the scene before the light falls incident uponthe image sensor; and an actuator configured to move one or both of theimage sensor and the lens, wherein the processor is configured withinstructions to cause the actuator to move the one or both of the imagesensor and the lens to an in-focus position based on the autofocusadjustment.
 3. The imaging system of claim 1, wherein, to determinewhether the phase difference exceeds the level of blurriness, theprocessor is configured with instructions to: determine a bounding phasedifference value corresponding to the level of blurriness; identify arange defined by the bounding phase difference value; determine whetherthe phase difference falls within the range; and in response todetermining that the phase difference falls within the range, calculatethe autofocus adjustment based on the phase difference.
 4. The imagingsystem of claim 1, wherein the processor is configured with instructionsto, in response to determining that the phase difference exceeds thelevel of blurriness, determine whether an autofocus process of theimaging system is in a monitoring state.
 5. The imaging system of claim4, wherein the processor is configured with instructions to: in responseto determining that the autofocus process is in the monitoring state,generate instructions that configure the plurality of diodes captureadditional phase detection information; determine an additional phasedifference between at least third and fourth images generated based onthe additional phase detection information; determine whether theadditional phase difference exceeds the level of blurriness; and inresponse to determining that the additional phase difference does notexceed the level of blurriness, calculate the autofocus adjustment basedon the additional phase difference.
 6. The imaging system of claim 4,wherein the processor is configured with instructions to: in response todetermining that the autofocus process is in not the monitoring state,evaluate a bounding phase difference value that corresponds to the levelof blurriness; clip the phase difference to the bounding phasedifference value; and calculate the autofocus adjustment based on thebounding phase difference value.
 7. The imaging system of claim 1,further comprising a masking element positioned above each of the secondsubset of the plurality of diodes to block a portion of lightpropagating from the target scene so each of the second subset of theplurality of diodes only collects the light propagating from the targetscene in a specific direction.
 8. The imaging system of claim 1, furthercomprising a microlens positioned above at least two adjacent diodes ofthe second subset of the plurality of diodes, the microlens formed topass a first portion of light propagating from the target scene to afirst of the at least two adjacent diodes and a second portion of lightpropagating from the target scene to a second of the at least twoadjacent diodes, the first portion propagating in a first direction andthe second portion propagating in a second direction.
 9. The imagingsystem of claim 1, wherein the plurality of diodes comprise asemiconductor substrate having a two-dimensional matrix of the pluralityof diodes.
 10. The imaging system of claim 1, wherein: optical elementsassociated with a first half of the second subset of the plurality ofdiodes are configured such that each diode of the first half of thesecond subset of the plurality of diodes only collects light propagatingfrom the scene in a first direction to generate a first half of thephase detection information; and optical elements associated with asecond half of the second subset of the plurality of diodes areconfigured such that each diode of the second half of the second subsetof the plurality of diodes only collects light propagating from thetarget scene in a second direction to generate a second half of thephase detection information.
 11. A process for phase detection autofocuscomprising: accessing image information from a plurality of imagingdiodes, the image information representing a scene; accessing phasedetection information received from a plurality of phase detectiondiodes; analyzing the image information to identify a level ofblurriness of a region of interest, determining a phase differencebetween at least first and second images generated based on the phasedetection information, determining whether the phase difference exceedsthe level of blurriness, and in response to determining that the phasedifference does not exceed the level of blurriness, calculating anautofocus adjustment based on the phase difference.
 12. The process ofclaim 11, further comprising generating instructions to positioncomponents of an imaging system in an in-focus position based on theautofocus adjustment.
 13. The process of claim 11, wherein, to determinewhether the phase difference exceeds the level of blurriness, theprocess further comprises: determining a bounding phase difference valuecorresponding to the level of blurriness; identifying a range defined bythe bounding phase difference value; determining whether the phasedifference falls within the range; and in response to determining thatthe phase difference falls within the range, calculating the autofocusadjustment based on the phase difference.
 14. The process of claim 11,further comprising, in response to determining that the phase differenceexceeds the level of blurriness, determining whether an autofocusprocess of the imaging system is in a monitoring state.
 15. The processof claim 14, further comprising: in response to determining that theautofocus process is in the monitoring state, generating instructionsthat configure the plurality of diodes capture additional phasedetection information; determining an additional phase differencebetween at least third and fourth images generated based on theadditional phase detection information; determining whether theadditional phase difference exceeds the level of blurriness; and inresponse to determining that the additional phase difference does notexceed the level of blurriness, calculating the autofocus adjustmentbased on the additional phase difference.
 16. The process of claim 15,further comprising: in response to determining that the autofocusprocess is in not the monitoring state, evaluating a bounding phasedifference value that corresponds to the level of blurriness; clippingthe phase difference to the bounding phase difference value; andcalculating the autofocus adjustment based on the bounding phasedifference value.
 17. An imaging apparatus comprising: means forcapturing image information representing a scene; means for capturingphase detection information; and means for analyzing the imageinformation to identify a level of blurriness of a region of interest;means for determining a phase difference between at least first andsecond images generated based on the phase detection information; meansfor determining whether the phase difference exceeds the level ofblurriness; and means for, in response to determining that the phasedifference does not exceed the level of blurriness, calculating anautofocus adjustment based on the phase difference.
 18. The apparatus ofclaim 17, further comprising means for, in response to determining thatthe phase difference exceeds the level of blurriness, determiningwhether an autofocus process of the imaging apparatus is in a monitoringstate.
 19. The apparatus of claim 18, further comprising: means for, inresponse to determining that the autofocus process is in the monitoringstate, generating instructions that configure the plurality of diodescapture additional phase detection information; means for determining anadditional phase difference between at least third and fourth imagesgenerated based on the additional phase detection information; means fordetermining whether the additional phase difference exceeds the level ofblurriness; and means for, in response to determining that theadditional phase difference does not exceed the level of blurriness,calculating the autofocus adjustment based on the additional phasedifference.
 20. The apparatus of claim 19, further comprising: meansfor, in response to determining that the autofocus process is in not themonitoring state, evaluating a bounding phase difference value thatcorresponds to the level of blurriness; means for clipping the phasedifference to the bounding phase difference value; and means forcalculating the autofocus adjustment based on the bounding phasedifference value.